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(54) Titie: COMPUTER-BASED PATIENT RECORD AND MESSAGE DELIVERY SYSTEM 
(57) Abstract 

A Computer-based Patient Recoixi (CPR) system includes user equipment devices which are configured for speech synthesis in response 
to speech markup language text and which are connected via a network to a middle tier of a server system. The CPR system further mcludes 
a mes^ge delivery facility for delivery of textual messages to any of pager, electronic mail, or voice mail (after text~to-speech synthesis) 
message delivery vehicles. The server system accesses a user specific data store containing speech synthesis profiles which include prosodic 
infomiation of the voices and speech of users, and message delivery profiles which specify which of the aforementioned message delivery 
vehicles arc to be used and in what oixler. The stored speech synthesis information associated with an ongmator of a message and Uie 
stored message delivery information associated with the recipient of message are provided by the seiver to user equipment or a reminder 
generator to produce speech markup files containing infomation needed to synthesize the vocal and speech chaiactenstics of the ongmator 
accompanied by delivery instructions reflecting the message delivery preferences of the recipient 
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COMPUTER-BASED PATIENT RECORD AND MESSAGE DELIVERY SYSTEM. 



The present invention relates generally to a computer system for enabling 
access of records in electronic form via a network, such as an intranet, which includes a 
facihty for delivery of messages. The invention further relates to a method for message 
dehvery. The invention is particularly useful in the medical domain in conjunction with 
5 Computer-based Patient Record (CPR) systems and in the provision of alerts to physicians and 
other health care professionals. 



CPR systems maintain multimedia patient records of one or more health care 
institutions such as histories, reports, charts, and images in electronic digital form in a server 

10 system so that authorized users may access the records remotely employing user equipment, 
such as desktop or portable computing devices, coupled to the server system via wired and/or 
wireless network paths. 

Speech interfaces for computer systems can employ speech recognition for 
command and control, thereby providing for hands-jfree navigation by the user. Also, useful 

1 5 • information can be optimally arranged on screen and the user can remain focussed thereon by 
eliminating distracting and space consuming Graphical User Interface (GUI) control objects 
such as buttons, hot spots and/or menus that are typically used with a mouse or other pointing 
device. Speech recognition for dictation can enable health care personnel to enter reports when 
their hands are busy or where touching a keyboard would be a hygiene risk. 

20 Text-to-speech synthesis can provide background notification of events and 

status changes, such as printer activity, by speech synthesized messages thereby avoiding 
visual distraction of the user. The use of synthesized speech for audible messages also 
minimizes storage requirements for the messages. 

A Java® Speech Application^Programming Interface (API) is substantially 

25 available to support speech synthesis, conmxand-and-control recognizers and dictation 
systems. The Java® Speech API includes a Java® Speech Markup Language (JSML) for 
speech synthesis and a Java® Speech Grammar Format (JSGF) for speech recognizers. JSGF 
provides mle-based grammar indicating possible alternative conunands, actions, objects, and 
their relative probabilities. JSML supports exphcit specifications for delimited words and 
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phrases such as structure (paragraph and sentence), special handling of dates and times, 
breaks, and prosodic information (speaking rate, volume, baseline pitch, pitch range). 

In hospital environments, health care personnel are not always at their user 
equipment devices. It is a drawback of the known system that it is not possible for health care 
personnel to issue alerts and reminders via the CPR system which can be received via a variety 
of message delivery vehicles, such as voice mail, pager and electronic mail. 

It is an object of the present invention to provide a message delivery facility in a 
computer network system which includes speech synthesis of text specified in a speech 
markup language, such as JSML. 

Thereto, a system according to the invention is defined in Claim 1 in this 
system the server system has access to a storage system containing user specific information 
including speech synthesis profiles of users of the system, each profile of a user including 
prosodic information for synthesizing vocal charactoistics of the user, such as baseline pitch 
and pitch range, and including message delivery profiles of users indicating which of a 
plurality of possible message delivery vehicles are to be tised, and in what order. The 
computer system has the capability of generating messages in the form of speech markup 
language text which includes prosodic information of a predetermined person associated with 
the message, usually the originator, to be initiated and supplied to a message delivery system 
for delivering the message to the recipient in accordance with the recipient's stored message 
delivery profile. 

It is a further object of the present invention that the text specified in speech 
markup language include speaker profile information so that the message may be personalized 
with prosodic characteristics of the voice of a predetermined person, for example the 
originator of the message, in order to invoke appropriate attention by the recipient. 

It is yet another object of the preserit invention that the computer system 
provide for delivery of messages in accordance with a message delivery profile of the recipient 
indicating which message delivery vehicles should be used and in what order. 

A fiirther advantage of the system according to the invention is that alerts and 
reminders when delivered as audible messages have appropriate clarity, structure, and 
prosodic characteristics, so they are easily understandable and receive attention appropriate to 
the nature of the information conveyed. 

The message delivery system includes a speech synthesizer for receiving the 
speech markup language text and synthesizing therefirom an audible message, which is similar 
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in vocal characteristics to the predeteimined person, and also appears clear and natural due to 
the use of delimiters in the markup language text specifying emphasis or volume, speaking 
rate, and items to be said as dates and times. The message delivery system includes as one of 
the message delivery vehicles, a voice mail facility which receives synthesized speech from 
5 the speech synthesizer. Other message delivery vehicles of the message delivery system 
include a pager system, and an electronic mail system. 

The invention further relates to a method for message delivery, a method 
according to the invention is defined in Claim 8. 

Other objects, features and advantages of the present invention will become 
10 apparent upon perasal of the following detailed description when taken in conjunction with the 
appended drawing, wherein: 

Figure 1 is a schematic diagram of a speech enabled CPR system in accordance 
with the present invention which is capable of generating alerts, reminders, and reports in the 
1 5 form of a speech markup language. 

Referring to Figure 1 of the drawing, there is shown a computer system 10 for 
delivery of Computer-based Patient Records (CPRs) to users at user equipment devices, or 
client computers 12, of which one is illustrated, from a server system generally comprising 

20 servers 14 and 16, via network paths such as 18. 

Server 14 may be one or a plurality of servers configured to together perform 
the functions of a Hypertext Transfer Protocol (HTTP) server and also a Common Object 
Request Broker Architecture (CORBA) server, using Intemet-Inter-Orb Protocol (HOP), 
which act as a middle tier between the user equipment devices or client computers 12 and the 

25 back end server 1 6 which is a Database Management System in conununication with the 

various information systems of the health care institution(s), including Picture Archiving and 
Communications System (PACS), Hospital Information System (HIS), Radiology Information 
System (RIS), laboratory computing system, and pharmacy computing system, in order to 
supply requested patient information objects to middle tier server 14. This middle tier server 

30 provides services to HTTP clients and CORBA clients via an intranet accessible via network 
paths 1 8. CPR information may be presented on the display 20 of the user equipment 1 2 
employing web-based tools such as Hyper Text Mark-up Language (HTML) pages and Java® 
(a trademark of Sun Microsystems, Inc.) applets that are downloaded to the user equipment by 
server 14 also acting as a Java® Object Request Broker (ORB), which downloaded applets 
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and run thereon inside a web browser. The browser/ Java® applet combination is represented 
by the element 22. 

User equipment 12, and the applets downloaded thereto, are preferably speech- 
enabled. Thus, user equipment 12 includes a speech synthesizer 24 which feeds a speaker 26, 
5 and a speech recognition engine or recognizer 28 which is fed by a microphone 30. However, 
the usual input devices, such as mouse and keyboard 25 should also be provided. As is usual, 
speech synthesizer 24 and speech recognition engine 28 may be reaUzed by sofbvare. 
Recognizer 28, which is preferably used both for command and control purposes and for 
dictation purposes, includes a lexicon and may be further constrained by a context appropriate 

10 to the subject or specialty of the user. For purposes of illustration, synthesizer 24 and 
recognizer 28 are shown separately coupled to applet/browser combination 22 via Java® 
Speech Application Programming Interface (API) 32. The connection 34 between recognition 
engine 28 and the Java® Speech API is shown as bi-directional, because grammar in Java® 
Speech Grammar Format (JSGF) is sent to the recognizer 28 for command and control 

15 recognition purposes. Preferably only the subset thereof applicable to the current screen 
display is sent from the applet to the recognizer to improve the recognition accuracy. 

In addition to command and control necessary to access patient records, users 
input text which may be medical reports of particular patients or messages, such as alerts, for 
particular recipients. This text may be inputted and placed into appropriate fields of an 

20 associated downloaded template by the user via the microphone 30 and speech recognition 
engine 28, via mouse and keyboard 25, or a combination thereof 

User equipment 12 also contains a Java® Speech Markup Language (JSML) 
converter 33 which is preferably implemented by software under control of the applet and is 
shown functionally in bi-directional communication with the browser/applet combination 22. 

25 Its purpose is to receive user inputted or application generated text and a template which is 
associated with the text or in which the text is embedded and to produce a JSML text file 
therefrom. The template, which has previously been downloaded from server 14, has fields or 
defined locations in which text is embedded or with which text is associated and respective 
JSML delimiters are associated in one to one coxrespondence with respective fields. 

30 Consequently, through the use of JSML delimiters, such things as volume or emphasis, 

speaking rate, pitch change, and whether text should be said as a date or said as a time can be 
individually specified for the text of each field. In accordance with the invention, the JSML 
converter 33 has the capabiUty of producing a JSML file in which prosodic information such 
as the baseline pitch and pitch range (and may include other information such as age and sex 
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of the speaker) which are taken from user specific speech synthesis profile information 
downloaded to user equipment 12 by server 14 are specified for the entire JSML file. Thus, the 
JSML file also contains information which allows a speech synthesizer, such as 24, to 
synthesize speech having the vocal characteristics of a predetemiined person associated with 
5 the text, such as the originator or author thereof The prosodic characteristics which can be 
specified to approximate the voice and speech of a predetemiined speaker are not necessarily 
limited to those specifically provided for in JSML as possible delimiters, since JSML may also 
contain native instructions to a specified speech recognizer. 

An application data store 34 and a user specific data store 36 are coupled to or 

10 accessible by server 14. Stores 36 may be implemented together in a single storage device or 
m a combination of such devices. Application data store 34 includes at area 38, Hypertext 
Markup Language (HTML) pages and templates for server 14 to form HTML pages in 
accordance with personalization inforaiation stored in user specific data store 36 at area 40. 
This personalization information includes the specialty or title and affiliation of the user, 

15 specifications of the user's equipment 12 including display type or resolution, and the limiting 
bandwidth in the network path 1 8 between server 14 and user equipment 12. After the user 
logs on to the system at user equipment 12 and presents his ID and password and requests 
patient formation or a list of patients, server 14 retrieves the personalization infomiation and 
uses it to generate HTML pages to be supplied to user equipment 12 which are sqjpropriate to 

20 the user, his relationship to the patient(s), and the capability of his equipment and network 
connection. The inclusion of such capabihty information alWs the server 14 to limit the use 
of or number of pixels in graphical objects in the HTML pages where the display is low 
resolution (or text only) or the limiting bandwidth includes a relatively low baud rate 
telephone or wireless path which would otherwise produce unacceptably long dovmload times 

25 for such objects. 

Application data store 34 also contains z^plets in area 42 and associated speech 
recognition grammar files in area 44 which are read by server 14 and dovmloaded to user 
equipment 12 when needed. Further, appUcation data store 34 includes alert, reminder and 
report templates at area 46 and the user specific data store includes speech synthesis profiles at 

30 area 48 which are used by JSML converter 33 to produce JSML text files which are 

personalized to the voice of a predetemiined speaker. The reminder templates and speech 
synthesis profiles are used by a reminder generator and JSML formatter 49. The latter device 
is configured to maintain a store of scheduled reminders fit>m particular originators to 
particular recipients at predeterauned times, having predetemiined text in fields of an 
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associated reminder template. At some time prior to the scheduled delivery time generator and 
fomiatter 49 produces a JSML text file therefrom in a similar manner as the JSML converter 
33 operates. Then, alert/reminder profile data stored in area 50 of specific data store is 
accessed by generator and fomiatter 49 via server 14 to detemiine which message delivery 
5 vehicles are to be used, their characteristics, and in which order they should be tried. 

Generator and formatter 49 uses this infomiation to produce delivery 
instructions which are sent along with the JSML text file to alert/reminder output devices 
system 51 via server 14. Output devices system includes a pager system 52 for delivery of 
plain text messages to designated pagers carried by users, an electronic mail system 54 which 

10 can be accessed by users from a variety of types of wired or wireless communications or 
computing devices (e.g. Personal Digital Assistants (PDAs), cellular or wireless local loop 
phones, screen telephones), and a voicemail facility 56. The latter includes a speech 
synthesizer 58 (including the necessary parts of Java® Speech API) and a conventional voice 
mail system, which users may access by telephone to hear audible messages. In the case of 

1 5 message delivery to electronic mail system 54, it is expected that the delivery instmctions will 
include whether or not the textual message should include JSML delimiters or whether they 
should be removed. 

In a similar maimer, when alerts are composed by a user at user equipment 12, 
server 14 will send to the user equipment an alert/reminder profile of the intended recipient so 
20 that delivery instructions may be produced by user equipment. The alert with accompanying 
delivery can be sent firom user equipment 12 to output device system 51 via server 14. 

It should now be appreciated that the objects of the present invention are 

satisfied. 

While the present invention has been described in particular detail, it should 
25 also be appreciated that numerous modifications are possible within the intended spirit and 
scope of the invention. 
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CLAIMS: 



1 . A computer system ( 1 0) comprising: 
a server system (14,16); 

a plurality of user equipment devices (12) coupled to the server system via a 

network (18); 

5 a storage system (36) accessible to the network contain user specific 

information including speech synthesis profiles of users of the system, each profile (48) of a 
user including prosodic information for synthesizing vocal characteristics of the user, and 
including message delivery preferences (150) of users indicating which of a plurahty of 
possible message delivery vehicles are to be used; 

10 means (33) at user equipment for an originator to initiate production of a 

message for a recipient, said message being in the form of speech markup language text which 
includes prosodic information of a predetermined person associated with the message; and 
a message dehvery system (51) for delivering the message to the recipient in accordance with 
the recipient's stored message delivery profile (50), 

15 

2. The system as claimed in Claim 1 , wherein the message delivery system 
includes a speech synthesizer (24) for receiving the speech markup language text and 
synthesizing therefirom an audible message, which is similar in vocal characteristics to the 
predetermined person. 

20 

3. The system as claimed in Claim 2, wherein the message delivery system 
includes as one of said message delivery vehicles, a voice mail facihty (56) which receives 
synthesized speech from said speech synthesizer. 

25 4. The system as claimed m Claim 3, wherein the message delivery system 

includes as another of said message delivery vehicles, a pager system (52). 

5. The system as claimed in Claim 4, wherein the message delivery system 

includes as another of said message delivery vehicles, an electronic mail system (54), 
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6. The system of Claim 1 , wherein said predeteraiined person is the originator of 

the message. 

5 7. The system of Claim 1 , wherein said prosodic inforaiation includes for each 

user, baseline pitch, and pitch range. 

8, A method for message delivery comprising: 

obtaining a filled-in message template comprising message text, specifying 
10 message content, and message delimiters, specifying characteristics for guiding subsequent 
speech synthesis, 

retrieving speech synthesis profile information for a predetermined person 
specifying prosodic characteristics for synthesizing the voice of the predetermined person, 
providing a message containing speech markup language text generated from 
15 said filled-in message template and said speech profile information, 

retrieving a message delivery profile of a recipient specifying which one or 
more of a plurality of message delivery vehicles, and their order, are to be used for delivery of 
said message to said recipient, and 

transmitting said message to said one or more of the plurality of message 
20 delivery vehicles. 

9. The method of claim 8 wherein said obtaining a filled-in message template 
comprises the further steps of: 

retrieving a message template comprising fields with delimiters for specifying 
25 characteristics guiding subsequent speech synthesis 
providing message text, and 

embedding said message text in said template to produce said filled-in message 

template. 

30 10. The method of claim 9 wherein said message text either is provided by user 

input, or is provided be application generation, or is predetermined. * 

1 1- The method of claim 8 wherein the method comprises a further step of 

receiving said message at said one or more of the plurality of message dehvery vehicles. 
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12. The method of claim 1 1 wherein at least one of said one or more of the plurality 
of message delivery vehicles comprises a speech synthesizer, and 

wherein said receiving said message at said one message delivery vehicle 
comprises synthesizing said message into an audible message for said recipient, 
5 whereby the audible message appears clear, natural and similar in vocal 

characteristics to the predetermined person due synthesis guidance by said speech profile 
information and by said message template delimiters. 

13. The method of claim 8 wherein said prosodic characteristics for a 
1 0 predetermined person include basehne pitch and pitch range. 

14. The method of claim 8 wherein said speech markup language is JAVA® speech 
markup language. 
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FIG. 1 



